It's challenging to run queries in a traditional relational database management system (RDBMS) if the RDBMS is used to store large amounts of data, which may include millions or even billions of entries. Environments generating this amount of data are not uncommon, such as an event management system continuously receiving events from many different sources.
Indexes may be created to improve query performance. However, indexing on large amounts of data for every column in the database can reduce query performance as queries may have to wait for indexing to complete. Also, indexing increases the amount of storage needed for the database due to the size of the indices for environments requiring storage of large amounts of data. Also, write operations take longer because the tables need to be re-indexed.
In addition, with such large amounts of data, reviewing query results can become unmanageable. A query may generate thousands or even millions of hits. It is often difficult for the user to parse such a large number of hits. In a typical situation, a first hundred hits are presented to the user. The user reviews the hits through a graphical user interface (GUI) and then clicks on a button to retrieve the next hundred hits and so on, which is a time consuming and inefficient process.